Auto range control for active illumination depth camera

ABSTRACT

A method and apparatus for auto range control are described. In one embodiment, the apparatus comprises a projector configured to project a sequence of light patterns on an object; a first camera configured to capture a sequence of images of the object illuminated with the projected light patterns; a controller coupled to the projector and first camera and operable to receive the sequence of images and perform range control by controlling power of the sequence of light patterns being projected on the object and exposure time of a camera based on information obtained from the sequence of images captured by the camera.

RELATED APPLICATION

This patent application is a continuation Application of and claims thebenefit of U.S. application Ser. No. 14/977,140, entitled “Auto RangeControl for Active Illumination Depth Camera,” filed Dec. 21, 2015,which is incorporated by reference in its entirety.

FIELD OF THE INVENTION

Embodiments of the present invention relate to the field of camerasystems that perform three-dimensional (3-D) measurements by use oftriangulation capturing one or more images with the assistance of one ormore patterns projected by a projector; more particularly, embodimentsof the present invention relate to performing auto range control of suchcamera systems.

BACKGROUND OF THE INVENTION

Stereo depth cameras are well-known and are often used to measure adistance from an object. One such measurement device includes aprojector and a camera and is often included in a coded-lightthree-dimensional (3D) camera system. In a coded-light three-dimensional(3D) camera, the projector projects a known pattern (e.g., a sequence ofbinary code patterns (e.g., vertical stripes)) on an object (e.g., ascene), and an image of the object upon which the image is projected iscaptured by the camera. From the captured images, depth information maybe determined. One technique for determining depth in such devices isthrough the use of triangulation from the known relative positions ofthe camera and a projector. Thus, images of objects are captured andmeasurements are taken to determine depth information.

Cameras in the 3D camera systems have limited depth dynamic range. Thedepth dynamic range is dictated by the receiver electro-optic path. Insituations of objects that are close to the camera with good albedoversus situations of far objects or poor albedo (reflectivity), thecamera doesn't function well since it is either saturated or has verylow signal-to-noise (SNR).

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the invention, which, however, should not be taken tolimit the invention to the specific embodiments, but are for explanationand understanding only.

FIG. 1 illustrates one embodiment of an active coded light triangulationsystem.

FIG. 2 is a flow diagram of one embodiment of a process for controllingthe depth dynamic range.

FIG. 3 is a flow diagram of one embodiment of the process forcontrolling the depth dynamic range by controlling the light power andthe exposure time.

FIG. 4 illustrates one embodiment of the auto gain and auto exposuredomain of operation.

FIG. 5 illustrates one embodiment of an example system.

FIG. 6 illustrates an embodiment of a computing environment capable ofsupporting the operations described herein.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

In the following description, numerous details are set forth to providea more thorough explanation of the present invention. It will beapparent, however, to one skilled in the art, that the present inventionmay be practiced without these specific details. In other instances,well-known structures and devices are shown in block diagram form,rather than in detail, in order to avoid obscuring the presentinvention.

The description may use the phrases “in an embodiment,” or “inembodiments,” which may each refer to one or more of the same ordifferent embodiments. Furthermore, the terms “comprising,” “including,”“having,” and the like, as used with respect to embodiments of thepresent disclosure, are synonymous.

The term “coupled with,” along with its derivatives, may be used herein.“Coupled” may mean one or more of the following. “Coupled” may mean thattwo or more elements are in direct physical, electrical, or opticalcontact. However, “coupled” may also mean that two or more elementsindirectly contact each other, but yet still cooperate or interact witheach other, and may mean that one or more other elements are coupled orconnected between the elements that are said to be coupled with eachother. The term “directly coupled” may mean that two or more elementsare in direct contact.

FIG. 1 illustrates one embodiment of an active coded light triangulationsystem. The system includes coded light range cameras operating byprojecting a sequence of one-dimensional binary (“black” and “white”)patterns onto a scene, such that the produced binary code encodes theangle of the projection plane. Depth is then reconstructed bytriangulation consisting of computing the intersection of an imaginaryray emanating from the camera with the plane emanating from theprojector.

Referring to FIG. 1, capture device 100 may include a 3D scanner, a 3Dcamera or any other device configured for a 3D object acquisition. Insome embodiments, as illustrated, capture device 100 includes an imagecapturing device 102 (e.g., a digital camera) and a projector unit 104,such as a laser projector or laser scanner, having a number ofcomponents. In some embodiments, digital camera 102 may comprise aninfrared (IR) camera, and the projector unit 104 may comprise an IRprojector.

Projector unit 104 is configured to project a light pattern as describedabove and may comprise a one-dimensional code projector. In oneembodiment, the light patterns comprise one-dimensional coded lightpatterns, e.g., the patterns that may be described by one-dimensional orlinear codes. The light patterns formed by the laser planes on a surfaceof the object may be received by image capturing device 102 and sensed(e.g., read) by a sensor of image capturing device 102. Based on thereadings of the multiple scans of the light patterns accumulated duringa sensing cycle of the sensor, capture device 100 may be configured toreconstruct the shape of the object.

In some embodiments, capture device 100 may further include anotherimage capturing device, such as digital camera 103. In some embodiments,digital camera 103 may have a resolution that is different than that ofdigital camera 103. For example, digital camera 102 may be amulti-chromatic camera, such as red, green, and blue (RGB) cameraconfigured to capture texture images of an object.

Capture device 100 may further include a processor 106 that may be inoperative communication with the image camera component 101 over a busor interconnect 107. Processor 106 may include a standardized processor,a specialized processor, a microprocessor, or the like that may executeinstructions that may include instructions for generating depthinformation, generating a depth image, determining whether a suitabletarget may be included in the depth image, or performing otheroperations described herein.

Processor 106 may be configured to reconstruct the object based on theimages captured by digital camera 102, for example, using geometrytechniques or other techniques used for 3D image reconstruction.Processor 106 may be further configured to dynamically calibrate capturedevice 100 to correct distortions in the reconstructed image of theobject that may be caused, for example, by various external factors(e.g., temperature).

Capture device 100 may further include a memory 105 that may store theinstructions that may be executed by processor 106, images or frames ofimages captured by the cameras, user profiles or any other suitableinformation, images, or the like. According to one example, memory 105may include random access memory (RAM), read only memory (ROM), cache,Flash memory, a hard disk, or any other suitable storage component. Asshown in FIG. 1, memory component 105 may be a separate component incommunication with the cameras 101 and processor 106. Alternatively,memory 105 may be integrated into processor 106 and/or the image capturecameras 101. In one embodiment, some or all of the components 102-106are located in a single housing.

Processor 105, memory 104, other components (not shown), image capturingdevice 102, and projector unit 104 may be coupled with one or moreinterfaces (not shown) configured to facilitate information exchangeamong the above-mentioned components. Communications interface(s) (notshown) may provide an interface for device 100 to communicate over oneor more wired or wireless network(s) and/or with any other suitabledevice. In various embodiments, capture device 100 may be included to orassociated with, but is not limited to, a server, a workstation, adesktop computing device, or a mobile computing device (e.g., a laptopcomputing device, a handheld computing device, a handset, a tablet, asmartphone, a netbook, ultrabook, etc.).

In one embodiment, capture device 100 is integrated into a computersystem (e.g., laptop, personal computer (PC), etc.). However, capturedevice 100 can be alternatively configured as a standalone device thatis couplable to such a computer system using conventional technologiesincluding both wired and wireless connections.

In various embodiments, capture device 100 may have more or lesscomponents, and/or different architectures. For example, in someembodiments, capture device 100 may include one or more of a camera, akeyboard, display such as a liquid crystal display (LCD) screen(including touch screen displays), a touch screen controller,non-volatile memory port, antenna or multiple antennas, graphics chip,ASIC, speaker(s), a battery, an audio codec, a video codec, a poweramplifier, a global positioning system (GPS) device, a compass, anaccelerometer, a gyroscope, and the like. In various embodiments,capture device 100 may have more or less components, and/or differentarchitectures. In various embodiments, techniques and configurationsdescribed herein may be used in a variety of systems that benefit fromthe principles described herein.

Capture device 100 may be used for a variety of purposes, including, butnot limited to, being part of a target recognition, analysis, andtracking system to recognize human and non-human targets in a capturearea of the physical space without the use of special sensing devicesattached to the subjects, uniquely identify them, and track them inthree-dimensional space. Capture device 100 may be configured to capturevideo with depth information including a depth image that may includedepth values via any suitable technique including, for example,triangulation, time-of-flight, structured light, stereo image, or thelike.

Capture device 100 may be configured to operate as a depth camera thatmay capture a depth image of a scene. The depth image may include atwo-dimensional (2D) pixel area of the captured scene where each pixelin the 2D pixel area may represent a depth value such as a distance in,for example, centimeters, millimeters, or the like of an object in thecaptured scene from the camera. In this example, capture device 100includes an IR light projector 404, an IR camera 102, and a visiblelight RGB camera 103 that are configured in an array.

Various techniques may be utilized to capture depth video frames. Forexample, capture device 100 may use structured light to capture depthinformation. In such an analysis, patterned light (i.e., light displayedas a known pattern such as a grid pattern or a stripe pattern) may beprojected onto the capture area via, for example, IR light projector104. Upon striking the surface of one or more targets or objects in thecapture area, the pattern may become deformed in response. Such adeformation of the pattern may be captured by, for example, he IR camera102 and/or the RGB camera 103 and may then be analyzed to determine aphysical distance from capture device 100 to a particular location onthe targets or objects.

Capture device 100 may utilize two or more physically separated camerasthat may view a capture area from different angles, to obtain visualstereo data that may be resolved to generate depth information. Othertypes of depth image arrangements using single or multiple cameras canalso be used to create a depth image.

Capture device 100 may provide the depth information and images capturedby, for example, IR camera 102 and/or the RGB camera 103, including askeletal model and/or facial tracking model that may be generated bycapture device 100, where the skeletal and/or facial tracking models,depth information, and captured images are used to, for example, createa virtual screen, adapt the user interface, and control an application.

In summary, capture device 100 may comprise a projector unit 104, adigital camera (e.g., IR camera) 102, another digital camera (e.g.,multi-chromatic camera) 103, and a processor (controller) configured tooperate capture device 100 according to the embodiments describedherein. However, the above assembly configuration is described forillustration purposes only, and is should not be limiting to the presentdisclosure. Various configurations of an assembly for a 3D objectacquisition may be used to implement the embodiments described herein.For example, an assembly for a 3D object acquisition configured toenable the reconstructed object distortion corrections may include threedigital cameras, two of which may be used to reconstruct a 3D image ofan object, and the third camera (e.g. with a resolution that isdifferent than those of the two cameras) may be used to capture imagesof the object in order to identify image distortions in thereconstructed object and to compensate for identified distortions.

Automatic Range Control for a Depth Camera

In one embodiment, the system includes a controller to control thecamera (e.g., IR camera) and the projector (e.g., IR projector) toperform range control. In one embodiment, the controller is part ofprocessor 106, working in conjunction with software stored in memory 105of the capture device of FIG. 1. In one embodiment, the controllercontrols both the power of the projected light from the projector andthe exposure time of the camera to perform range control. In oneembodiment, the control of the power of the projected light from theprojector and the exposure time of the camera is based on the sceneanalysis. In this case, the scene analysis involves determining theamount of light that is being reflected back from an object in thescene, as opposed to analyzing content in the scene. In one embodiment,the control results in an enlarged depth dynamic range in the near andfar ends of the range. In one embodiment, in the near range, using thetechniques disclosed herein, the dynamic range increases from 25 cm to 3cm, while in the far end, the dynamic range goes from 70 cm to 1.8 m.

By observing the statistics of the imaged object, the controller controlthe exposure of the camera and/or the light (laser) power of theprojector in order to avoid camera saturation at close range (e.g., lessthan 30 cm) of highly reflective objects at one near end of the dynamicrange, while still being able to capture low reflective objects at alarge distance at the other far end of the dynamic range. Thesetechniques result in an increased range and quality of the digitalcaptured image. The increased quality enables increased accuracy ofdepth profiles and allows the system to adapt itself to the operationenvironment to produce the best possible results.

FIG. 2 is a flow diagram of one embodiment of a process for controllingthe depth dynamic range. The process is performed by processing logicthat may comprise hardware (circuitry, dedicated logic, etc.), software(such as is run on a general purpose computer system or a dedicatedmachine), firmware, or a combination of the three.

Referring to FIG. 2, the process begins by processing logic projecting asequence of light patterns on an object (processing block 201). In oneembodiment, the processing logic is part of a projector. In anotherembodiment, the processing logic is part of a controller sends controlsignals to the projector to cause the projector to project light ontothe object.

Next, processing logic captures a sequence of images of the objectilluminated with the projected light patterns using a camera (processingblock 202). In one embodiment, the processing logic is part of acontroller sends control signals to the camera to cause the camera tocapture the sequence of images of the illuminated object.

Using the captured sequence of images, processing logic controls thedepth dynamic range of the camera by controlling the power of thesequence of light patterns being projected on the object by theprojector and the exposure time of a camera (processing block 203). Inone embodiment, information is obtained from the captured sequence ofimages for use in controlling the depth dynamic range. In oneembodiment, the information comprises intensity information associatedwith a sequence of images captured by the camera. In another embodiment,the intensity information comprises an average intensity of a differencebetween intensity of a sequence of images captured by the camera.

In one embodiment, the controller obtains its statistics from a sequenceof captured images. For example, in one embodiment, the statistics arefrom the n-1 sequence (e.g., the previous, or last, sequence of capturedimages) including: an image captured while the active light is on (e.g.,the laser is on) referred to herein as I_1 and an image captured whenthe active light is off (e.g., the laser is off) referred to herein asI_0. Using these two images, the controller generates another image. Inone embodiment, this additional image is an icon image and is generatedby subtracting the image captured while the active light is off from theimage captured when the active light is on (i.e., Ic=I_1−I_0). In oneembodiment, each of I_1 and I_0 is a matrix of the image, which eachentry in the matrix being the value of the illumination of a pixel forthe image. These values in corresponding locations in two matrices aresubtracted from each other to obtain the icon image (Ic).

After obtaining the icon image, the controller determines an averageintensity value for the icon image. In one embodiment, the averageintensity value is determined by averaging all of the intensity valuesin the icon image (e.g., the intensity values in the icon image matrix).In another embodiment, he average intensity value is determined byaveraging a subset of the intensity values of the icon image (e.g., theintensity values in the icon image matrix). The average intensity valueof the icon image Ic, or power score, is denoted as Ps herein.

Using the power score, in one embodiment, the controller controls thedepth dynamic range according to the following action rules:

If (Ps > Threshold 1)  {  if exposure is minimal (e.g., the exposure hasreached a preset minimal value), then decrease laser power  elsedecrease camera exposure  } If (Ps > Threshold 2) {  if laser power ismaximal (e.g., the laser power has reached a preset maximum value), thenincrease exposure  else increase laser power  }

In one embodiment, Threshold 1 is 1250 and Threshold 2 is 650. In oneembodiment, these thresholds are learned statistically on several unitsand are dependent on the range of exposure values required and theeffective Z field of view (FOV). For short range applications, in oneembodiment, Threshold 1 is increased and Threshold 2 is decreased, andvice versa for far range applications.

In one embodiment, the laser gain increases and decreases are stepped atfive percent of the range, while exposure increases and decreases aremade at steps of 0.2 ms. Note that other step sizes may be used and theincrease and decrease step sizes need not be the same. Note that in oneembodiment, the increases and decreases are made based on recognizingthat light follows an inverse-square law in that the intensity isinversely proportional to the square of the distance from the source ofthat physical quantity. Thus, in such cases, decreases and increaseschange quadratically, as opposed to following a linear path.

FIG. 3 is a flow diagram of one embodiment of the process forcontrolling the depth dynamic range described above. The process isperformed by processing logic that may comprise hardware (circuitry,dedicated logic, etc.), software (such as is run on a general purposecomputer system or a dedicated machine), firmware, or a combination ofthe three.

Referring to FIG. 3, the controller compares the power score to a pairof thresholds, a first threshold and a second threshold, and controlsdepth dynamic range by:

decreasing the projector laser power if average intensity between asequence of images captured by the camera is above a first threshold andthe camera exposure is set to a preset minimal value;

decreasing the exposure of the camera if the average intensity between asequence of images captured by the camera is above the first thresholdand the camera exposure is not set to the preset minimal value;

increasing the camera exposure of the camera if average intensitybetween a sequence of images captured by the camera is below a secondthreshold and the projector laser power is set to a preset minimalvalue; and

decreasing the projector laser power if average intensity between asequence of images captured by the camera is below a second thresholdand the projector laser power is not set to the preset minimal value.

Example Operational Modes

In one embodiment, the controller the camera and projector using anumber of different operations modes. These operational modes include anautomatic (auto) laser gain mode, automatic (auto) exposure time mode,and auto laser gain with auto exposure time mode. The auto laser gainwith auto exposure time mode controls the depth dynamic range of thecamera by controlling the power of the sequence of light patterns beingprojected on the object by the projector and the exposure time of acamera using a number of operation modes. Each of these modes isdiscussed in more detail below.

Auto Laser Gain

When enabled, the controller decreases the laser gain if an object isclose (or moves close) to the camera sensor to avoid saturation andincreases the laser gain when the object moves away from the camerasensor. In one embodiment, the controller has a default control settingto have the laser gain always be at the highest laser gain possible.

Auto Exposure Time

When enabled, in one embodiment, the controller changes the exposuretime automatically when the object gets far from the camera sensor toincrease the dynamic range of the camera.

Auto Laser Gain with Auto Exposure

When both features are enabled, the controller activates the auto lasergain when the object intensity rises. When the object intensitydecreases, the controller increases the laser gain until it reaches themaximal value (e.g., preset maximum value). When the laser gain reachesits maximal value, the controller activates the auto exposure andincreases the exposure as required. When the object intensity increasesagain, the controller decreases the exposure until a minimal value(e.g., a preset minimum value) and activates the auto laser gain and soforth. FIG. 4 illustrates one embodiment of the auto gain and autoexposure domain of operation. Referring to FIG. 4, the object intensity(power level) is shown with a point in the middle of the objectintensity bar being the point where MsvR (Motion vs Range trade off, orexposure time) is equal to zero and the laser gain is at its maximumvalue. To one side (left) of this point, the controller uses auto lasergain control, while on the other side of the point (right), thecontroller use auto exposure control. In other words, in one embodiment,on one side of the point, the controller can only decrease laser while,while on the other side, the controller can only increase the exposuretime.

An Example of a Control Algorithm Algorithm Input

In one embodiment, the control algorithm uses an iconed version of theIR image (e.g., averaged and subsampled intensity IR image). In anotherembodiment, the control algorithm uses an IR image (created from theprojector of the capture device).

Algorithm Output

In one embodiment, the algorithm outputs a decision to change either thelaser power or the exposure. In one embodiment, the output indication tochange the power is as follows:

-   −1: decrease the power-   0: do not change the power-   1: increase the power

According to the operation mode and the current power level, thecontroller changes the laser power or the exposure control.

Configuration

One embodiment of the parameters and their default values are givenbelow. In one embodiment, these parameters are configurable.

Parameter Name Default Description LowerTh 0.1 The lower threshold belowwhich the power level is increased UpperTh 0.3 The upper threshold abovewhich the power level is descreased WindowSize 3 The size of the averagewindow - the number of power scores to accumulate in the averageZeroTrendFrameCount 20 The number of frames in the same power level tostart the trend to zero after. ZeroTrendMargin 0.01 The margin from thethresholds that below it no zero trend is allowed PowerLevelThSlope 0The slope of the threshold as a function of the power levelrelativeIRwindowSize 11 The size of the relative I1 decision, whichcontrols the jitteriness of switching from I1 to relative I1 and backI0MeanTh 5 The mean of I0 above which the ambient light overflow mightoccur I1I0RatioTh 1.25 The ration between I1 and I0 below which ambientlight overflow might occur

Power Score Calculation

In one embodiment, to begin the algorithm, the “power score” iscalculated. In one embodiment, the power score is calculated as follows.For each bin in a histogram calculated by the controller on a selectedregion of interest (ROI) in the icon image, the sum is calculated bymultiplying the number of pixels in the bin by the bin index, and thenthat product is divided by the number of pixels multiplied by themaximum possible bin index (e.g., 255). In one embodiment, the followingformula calculates the “power score” (average intensity [0,1]):

${PowerScore} = {\frac{\sum_{i = 0}^{255}\left( {{{Hist}\lbrack i\rbrack}*i} \right)}{255*{\sum_{i = 0}^{255}\left( {{Hist}\lbrack i\rbrack} \right)}} \in \left\lbrack {0,1} \right\rbrack}$

One Embodiment of the Control Algorithm

The following pseudo code describes one embodiment of the controlalgorithm. In one embodiment, these operations are performed by thecontroller.

-   1. Calculate power score of I0->I0 Score-   2. Calculate power score of I1->I1 Score-   3. Calculate power score ->Score=I1Score−I0Score

The first three operations above determine the power score that is usedto control the algorithm.

-   4. Update scoreswindow ->ScoresWindow [frameIndex mod    windowSize]=Score

In one embodiment, the power scores are aggregated over consecutiveimage captures in time to determine if there is a trend. In oneembodiment, the window size (windowSize) is usually 1-3. In oneembodiment, this is configurable. The scores are aggregated into anarray.

-   5. Calculate relative IRstatus->-   a. If I0Score>I0MeanTh/255 AND (I1Score/I0Score)<I1I0RatioTh

This condition determines if there is too much ambient light in theimage: the dark image is to bright and the ration between the bright anddark images it too close to one according to one embodiment.

relativeIRstatus=0

-   b. else

relativeIRstatus=1

-   6. Update Relative IR status window ->-   RelativeIRWindow[I mod relativeIRwindowSize]=relativeIRstatus

In one embodiment, this operation determines the amount of ambient lightin the scene. If the amount of ambient light is too big, then the statusis set to 1. If the amount of ambient light is too low, the status isset to 0. This is track so that in the case the ambient light is toobig, then increasing the laser power and/or increasing the exposure isnot performed to avoid increasing the amount of saturation. Note that inone embodiment, this is tracked over a number of consecutive imagecaptures using a window (e.g., ten samples) to reduce the likelihood ofnoise influencing the process.

-   7. If RelativeIRWindow is full

If the RelativeIRWindow is full, then there is enough samples toproceed.

-   a. AvgStatus=Average (RelativeIRWindow)

The average of the values in the RelativeIRWindow is determined.

-   b. If AvgStatus>0.8

newRelativeIRStatus=1

-   Else if AvgStatus<0.2

newRelativeIRStatus=0

In one embodiment, if the average (AvgStatus) is greater than 0.8, thenthere is too much ambient light, while if the average is less than 0.2,then there isn't too much ambient light.

-   8. If ScoresWindow is full

If the ScoresWindow is full, then there are enough samples to proceed.

-   a. Calculate average score in the window

->AvgScore=Average (ScoresWindow)

If there are enough samples, then the average score (AvgScore) of theintensity values in the ScoresWindow is calculated. Based on thataverage score and its relation to two thresholds, the controllercontrols the depth dynamic range using 8b-d below.

-   b. If AvgScore>UpperTh+PowerLevelThSlope*CurrPowerLevel

i. NextPowerLevel=CurrPowerLevel−1;

ii. empty ScoresWindow

iii. LastFixIdx=0

If the average score of the intensity values is greater than the sum ofupper threshold (above which the power level is decreased) and theproduct of the slope of the threshold as a function of the power leveland the current power level, then the current power level is reduced by1, the window of intensity values is cleared and a variable trackingwhen the last change was made by the controller is reset (e.g., set tozero). Note that the slope is meant to make it harder to cross reachdistant power levels by increasing the threshold that the power score(Ps) should cross in order to get further from the target power level.In one embodiment, the slope is configurable, and can be 0 (unused).

-   c. ifAvgScore<LowerTh+PowerLevelThSlope*CurrPowerLevel

i. NextPowerLevel=CurrPowerLevel+1

ii. empty ScoresWindow

iii. LastFixIdx=0

If the average score of the intensity values is less than the sum oflower threshold (below which the power level is increased) and theproduct of the slope of the threshold as a function of the power leveland the current power level, then the current power level is increasedby 1, the window of intensity values is cleared and a variable trackingwhen the last change was made by the controller is reset (e.g., set tozero).

-   d. else (AvgScore between the margins)

i. LastFixIdx++

ii. If (LastFixIdx>ZeroTrendFrameCount)

AND

-   {((power level >0) and AvgScore>LowerTh+ZeroTrendMargin)

OR

-   (power level<0) and AvgScore<UpperTh−ZeroTrendMargin)}-   Then NextPowerLevel=CurrPowerLevel−Sign (CurrPowerLevel)

If the average score of the intensity values is between the thresholds,then the controller doesn't change the power level, the window ofintensity values is not cleared and a variable tracking when the lastchange was made by the controller is incremented (e.g., incremented byone).

In one embodiment, if the value of the variable tracking when the lastchange was made exceeds the zero trend frame count, indicating that thepower level has remained the same for a number of frames and the powerlevel and based on the power level and the relation between the averagescore of the intensity values and both the lower and upper thresholds,then the controller than calculates the next power level based on adifference between the current power level and the sign of the currentpower level. This recognizes that when the power level has remained thesame for a certain number of frames, then the controller is able to moveback to a nominal power level that the camera system is selected to workat.

-   9. If relativeIRstatus==0

Set powerLevel to zero state

-   Else

Return (CurrPowerLevel-NextPowerLevel)

In one embodiment, if there is too much ambient light (as indicated byrelativeIR status being set to 0), then the power level is set to zero,which means that there is minimal exposure time (e.g., exposure time isat a preset minimum value) and the laser power is set to its maximumvalue (FIG. 4). If there is not too much ambient light, the next powerlevel is set to the difference between the values of the current powerlevel and next power level.

Thus, in one embodiment, the “nextPowerLevel” indicates whether toincrease or decrease the power. Once this is set, the decision aboutwhether to change the laser power or the exposure is made, dependent onthe current setting, according to the graph shown in FIG. 4.

In one embodiment, the range control is implemented in the camera deviceitself. In another embodiment, this is implemented in a host. There areadvantages to close this kind of control loop in a lower level (devicelevel).

FIG. 5 illustrates one embodiment of an example system 600 having one ormore processor(s) 604, system control module 608 coupled to at least oneof the processor(s) 604, system memory 612 coupled to system controlmodule 608, non-volatile memory (NVM)/storage 614 coupled to systemcontrol module 608, and one or more communications interface(s) 620coupled to system control module 608. In some embodiments, the system600 may include capture device 100 and provide logic/module thatperforms functions aimed at depth and texture calibration, along withdepth reconstruction and other functions, described herein.

In some embodiments, the system 600 may include one or morecomputer-readable media (e.g., system memory or NVM/storage 614) havinginstructions and one or more processors (e.g., processor(s) 604) coupledwith the one or more computer-readable media and configured to executethe instructions to implement a module to perform depth and texturecalibration, along with depth reconstruction and other functions,described herein.

System control module 608 for one embodiment may include any suitableinterface controllers to provide for any suitable interface to at leastone of the processor(s) 604 and/or to any suitable device or componentin communication with system control module 608.

System control module 608 may include memory controller module 610 toprovide an interface to system memory 612. The memory controller module610 may be a hardware module, a software module, and/or a firmwaremodule. System memory 612 may be used to load and store data and/orinstructions, for example, for system 600. System memory 612 for oneembodiment may include any suitable volatile memory, such as suitableDRAM, for example. System control module 608 for one embodiment mayinclude one or more input/output (I/O) controller(s) to provide aninterface to NVM/storage 614 and communications interface(s) 620.

The NVM/storage 614 may be used to store data and/or instructions, forexample. NVM/storage 614 may include any suitable non-volatile memory,such as flash memory, for example, and/or may include any suitablenon-volatile storage device(s), such as one or more hard disk drive(s)(HDD(s)), one or more compact disc (CD) drive(s), and/or one or moredigital versatile disc (DVD) drive(s), for example. The NVM/storage 614may include a storage resource physically part of a device on which thesystem 600 is installed or it may be accessible by, but not necessarilya part of, the device. For example, the NVM/storage 614 may be accessedover a network via the communications interface(s) 620.

Communications interface(s) 620 may provide an interface for system 600to communicate over one or more network(s) and/or with any othersuitable device. The system 600 may wirelessly communicate with the oneor more components of the wireless network in accordance with any of oneor more wireless network standards and/or protocols.

For one embodiment, at least one of the processor(s) 604 may be packagedtogether with logic for one or more controller(s) of system controlmodule 608, e.g., memory controller module 610. For one embodiment, atleast one of the processor(s) 604 may be packaged together with logicfor one or more controllers of system control module 608 to form aSystem in Package (SiP). For one embodiment, at least one of theprocessor(s) 604 may be integrated on the same die with logic for one ormore controller(s) of system control module 608. For one embodiment, atleast one of the processor(s) 604 may be integrated on the same die withlogic for one or more controller(s) of system control module 608 to forma System on Chip (SoC).

In various embodiments, the system 600 may have more or less components,and/or different architectures. For example, in some embodiments, thesystem 600 may include one or more of a camera, a keyboard, liquidcrystal display (LCD) screen (including touch screen displays),non-volatile memory port, multiple antennas, graphics chip,application-specific integrated circuit (ASIC), and speakers.

In various implementations, the system 600 may be, but is not limitedto, a mobile computing device (e.g., a laptop computing device, ahandheld computing device, a tablet, a netbook, etc.), a laptop, anetbook, a notebook, an ultrabook, a smartphone, a tablet, a personaldigital assistant (PDA), an ultra mobile PC, a mobile phone, a desktopcomputer, a server, a printer, a scanner, a monitor, a set-top box, anentertainment control unit, a digital camera, a portable music player,or a digital video recorder. In further implementations, the system 600may be any other electronic device.

FIG. 6 illustrates an embodiment of a computing environment 700 capableof supporting the operations discussed above. The modules describedbefore can use the depth information (e.g., values) and other datadescribed above to perform these functions. The modules and systems canbe implemented in a variety of different hardware architectures and formfactors.

Command Execution Module 701 includes a central processing unit to cacheand execute commands and to distribute tasks among the other modules andsystems shown. It may include an instruction stack, a cache memory tostore intermediate and final results, and mass memory to storeapplications and operating systems. Command Execution Module 701 mayalso serve as a central coordination and task allocation unit for thesystem.

Screen Rendering Module 721 draws objects on the one or more multiplescreens for the user to see. It can be adapted to receive the data fromVirtual Object Behavior Module 704, described below, and to render thevirtual object and any other objects and forces on the appropriatescreen or screens. Thus, the data from Virtual Object Behavior Module704 would determine the position and dynamics of the virtual object andassociated gestures, forces and objects, for example, and ScreenRendering Module 721 would depict the virtual object and associatedobjects and environment on a screen, accordingly. Screen RenderingModule 721 could further be adapted to receive data from Adjacent ScreenPerspective Module 707, described below, to either depict a targetlanding area for the virtual object if the virtual object could be movedto the display of the device with which Adjacent Screen PerspectiveModule 707 is associated. Thus, for example, if the virtual object isbeing moved from a main screen to an auxiliary screen, Adjacent ScreenPerspective Module 707 could send data to the Screen Rendering Module721 to suggest, for example in shadow form, one or more target landingareas for the virtual object on that track to a user's hand movements oreye movements.

Object and Gesture Recognition System 722 may be adapted to recognizeand track hand and harm gestures of a user. Such a module may be used torecognize hands, fingers, finger gestures, hand movements and a locationof hands relative to displays. For example, Object and GestureRecognition System 722 could for example determine that a user made abody part gesture to drop or throw a virtual object onto one or theother of the multiple screens, or that the user made a body part gestureto move the virtual object to a bezel of one or the other of themultiple screens. Object and Gesture Recognition System 722 may becoupled to a camera or camera array, a microphone or microphone array, atouch screen or touch surface, or a pointing device, or some combinationof these items, to detect gestures and commands from the user.

The touch screen or touch surface of Object and Gesture RecognitionSystem 722 may include a touch screen sensor. Data from the sensor maybe fed to hardware, software, firmware or a combination of the same tomap the touch gesture of a user's hand on the screen or surface to acorresponding dynamic behavior of a virtual object. The sensor date maybe used to momentum and inertia factors to allow a variety of momentumbehavior for a virtual object based on input from the user's hand, suchas a swipe rate of a user's finger relative to the screen. Pinchinggestures may be interpreted as a command to lift a virtual object fromthe display screen, or to begin generating a virtual binding associatedwith the virtual object or to zoom in or out on a display. Similarcommands may be generated by Object and Gesture Recognition System 722,using one or more cameras, without the benefit of a touch surface.

Direction of Attention Module 723 may be equipped with cameras or othersensors to track the position or orientation of a user's face or hands.When a gesture or voice command is issued, the system can determine theappropriate screen for the gesture. In one example, a camera is mountednear each display to detect whether the user is facing that display. Ifso, then the direction of attention module information is provided toObject and Gesture Recognition Module 722 to ensure that the gestures orcommands are associated with the appropriate library for the activedisplay. Similarly, if the user is looking away from all of the screens,then commands can be ignored.

Device Proximity Detection Module 725 can use proximity sensors,compasses, GPS (global positioning system) receivers, personal areanetwork radios, and other types of sensors, together with triangulationand other techniques to determine the proximity of other devices. Once anearby device is detected, it can be registered to the system and itstype can be determined as an input device or a display device or both.For an input device, received data may then be applied to Object Gestureand Recognition System 722. For a display device, it may be consideredby Adjacent Screen Perspective Module 707.

Virtual Object Behavior Module 704 is adapted to receive input fromObject Velocity and Direction Module 703, and to apply such input to avirtual object being shown in the display. Thus, for example, Object andGesture Recognition System 722 would interpret a user gesture and bymapping the captured movements of a user's hand to recognized movements,Virtual Object Tracker Module 706 would associate the virtual object'sposition and movements to the movements as recognized by Object andGesture Recognition System 722, Object and Velocity and Direction Module703 would capture the dynamics of the virtual object's movements, andVirtual Object Behavior Module 704 would receive the input from Objectand Velocity and Direction Module 703 to generate data that would directthe movements of the virtual object to correspond to the input fromObject and Velocity and Direction Module 703.

Virtual Object Tracker Module 706 on the other hand may be adapted totrack where a virtual object should be located in three-dimensionalspace in a vicinity of a display, and which body part of the user isholding the virtual object, based on input from Object Gesture andRecognition System 722. Virtual Object Tracker Module 706 may forexample track a virtual object as it moves across and between screensand track which body part of the user is holding that virtual object.Tracking the body part that is holding the virtual object allows acontinuous awareness of the body part's air movements, and thus aneventual awareness as to whether the virtual object has been releasedonto one or more screens.

Gesture to View and Screen Synchronization Module 708, receives theselection of the view and screen or both from Direction of AttentionModule 723 and, in some cases, voice commands to determine which view isthe active view and which screen is the active screen. It then causesthe relevant gesture library to be loaded for Object and GestureRecognition System 722. Various views of an application on one or morescreens can be associated with alternative gesture libraries or a set ofgesture templates for a given view.

Adjacent Screen Perspective Module 707, which may include or be coupledto Device Proximity Detection Module 725, may be adapted to determine anangle and position of one display relative to another display. Aprojected display includes, for example, an image projected onto a wallor screen. The ability to detect a proximity of a nearby screen and acorresponding angle or orientation of a display projected therefrom mayfor example be accomplished with either an infrared emitter andreceiver, or electromagnetic or photo-detection sensing capability. Fortechnologies that allow projected displays with touch input, theincoming video can be analyzed to determine the position of a projecteddisplay and to correct for the distortion caused by displaying at anangle. An accelerometer, magnetometer, compass, or camera can be used todetermine the angle at which a device is being held while infraredemitters and cameras could allow the orientation of the screen device tobe determined in relation to the sensors on an adjacent device. AdjacentScreen Perspective Module 707 may, in this way, determine coordinates ofan adjacent screen relative to its own screen coordinates. Thus, theAdjacent Screen Perspective Module may determine which devices are inproximity to each other, and further potential targets for moving one ormore virtual object's across screens. Adjacent Screen Perspective Module707 may further allow the position of the screens to be correlated to amodel of three-dimensional space representing all of the existingobjects and virtual objects.

Object and Velocity and Direction Module 703 may be adapted to estimatethe dynamics of a virtual object being moved, such as its trajectory,velocity (whether linear or angular), momentum (whether linear orangular), etc. by receiving input from Virtual Object Tracker Module706. The Object and Velocity and Direction Module 703 may further beadapted to estimate dynamics of any physics forces, by for exampleestimating the acceleration, deflection, degree of stretching of avirtual binding, etc. and the dynamic behavior of a virtual object oncereleased by a user's body part. Object and Velocity and Direction Module703 may also use image motion, size and angle changes to estimate thevelocity of objects, such as the velocity of hands and fingers

Momentum and Inertia Module 702 can use image motion, image size, andangle changes of objects in the image plane or in a three-dimensionalspace to estimate the velocity and direction of objects in the space oron a display. Momentum and Inertia Module 702 is coupled to Object andGesture Recognition System 722 to estimate the velocity of gesturesperformed by hands, fingers, and other body parts and then to applythose estimates to determine momentum and velocities to virtual objectsthat are to be affected by the gesture.

3D Image Interaction and Effects Module 705 tracks user interaction with3D images that appear to extend out of one or more screens. Theinfluence of objects in the z-axis (towards and away from the plane ofthe screen) can be calculated together with the relative influence ofthese objects upon each other. For example, an object thrown by a usergesture can be influenced by 3D objects in the foreground before thevirtual object arrives at the plane of the screen. These objects maychange the direction or velocity of the projectile or destroy itentirely. The object can be rendered by the 3D Image Interaction andEffects Module 705 in the foreground on one or more of the displays.

In a first example embodiment, an apparatus comprises a projectorconfigured to project a sequence of light patterns on an object, a firstcamera configured to capture a sequence of images of the objectilluminated with the projected light patterns, and a controller coupledto the projector and first camera and operable to receive the sequenceof images and perform range control by controlling power of the sequenceof light patterns being projected on the object and exposure time of acamera based on information obtained from the sequence of imagescaptured by the camera.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the information comprisesintensity information associated with a sequence of images captured bythe camera. In another example embodiment, the subject matter of thisexample embodiment can optionally include that the intensity informationcomprises an average intensity of a difference between intensity of asequence of images captured by the camera.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the controller is operable tocontrol depth dynamic range by decreasing the projector power if averageintensity between a sequence of images captured by the camera is above afirst threshold and the camera exposure is set to a preset minimal valueand decreasing the exposure of the camera if the average intensitybetween a sequence of images captured by the camera is above the firstthreshold and the camera exposure is not set to the preset minimalvalue.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the controller is operable tocontrol depth dynamic range by increasing the camera exposure of thecamera if average intensity between a sequence of images captured by thecamera is below a second threshold and the projector power is set to apreset minimal value and decreasing the projector power if averageintensity between a sequence of images captured by the camera is below asecond threshold and the projector laser power is not set to the presentminimal value.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the controller is operable tocontrol power of the sequence of light patterns being projected on theobject and exposure time of a camera based on information obtained fromthe sequence of images captured by the camera by performing automaticprojector gain and automatic exposure control based on object intensity.In another example embodiment, the subject matter of this exampleembodiment can optionally include that performing automatic projectorgain and automatic exposure control comprises activating automaticprojector gain when object intensity associated with the objectincreases and increasing projector gain as the object intensitydecreases until the object intensity reaches a preset maximum value, andwhen the object intensity reaches the preset maximum value, thenactivating the automatic exposure control until a preset minimum valueis reached after which time the automatic projector gain is activated.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the processing unit reconstructsdepth using triangulation.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the projector comprises aninfrared (IR) projector, the first camera comprises an IR camera, andthe second camera comprises a red, green, and blue (RGB) camera.

In a second example embodiment, a method comprises projecting a sequenceof light patterns on an object using a projector, capturing a sequenceof images of the object illuminated with the projected light patternsusing a camera, and controlling depth dynamic range by controlling powerof the sequence of light patterns being projected on the object andexposure time of a camera based on information obtained from thesequence of images captured by the camera.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that the information comprisesintensity information associated with a sequence of images captured bythe camera. In another example embodiment, the subject matter of thisexample embodiment can optionally include that the intensity informationcomprises an average intensity of a difference between intensity of asequence of images captured by the camera.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that controlling depth dynamic rangecomprises decreasing the projector laser power if average intensitybetween a sequence of images captured by the camera is above a firstthreshold and the camera exposure is set to a preset minimal value anddecreasing the exposure of the camera if the average intensity between asequence of images captured by the camera is above the first thresholdand the camera exposure is not set to the preset minimal value.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that controlling depth dynamic rangecomprises increasing the camera exposure of the camera if averageintensity between a sequence of images captured by the camera is below asecond threshold and the projector power is set to a preset minimalvalue and decreasing the projector power if average intensity between asequence of images captured by the camera is below a second thresholdand the projector laser power is not set to the present minimal value.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that controlling depth dynamic rangeby controlling power of the sequence of light patterns being projectedon the object and exposure time of a camera based on informationobtained from the sequence of images captured by the camera comprisesperforming automatic projector gain and automatic exposure control basedon object intensity. In another example embodiment, the subject matterof this example embodiment can optionally include that performingautomatic projector gain and automatic exposure control comprisesactivating automatic projector gain when object intensity associatedwith the object increases and increasing projector gain as the objectintensity decreases until the object intensity reaches a preset maximumvalue, and when the object intensity reaches the preset maximum value,then activating the automatic exposure control until a preset minimumvalue is reached after which time the automatic projector gain isactivated.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that the power of the sequence oflight patterns comprises laser power and the projector comprises aninfrared (IR) projector.

In a third example embodiment, an article of manufacture has one or morenon-transitory computer readable storage media storing instructionswhich when executed by a system to perform a method comprising:capturing, using a camera, a sequence of images of the objectilluminated with projected light patterns on an object using aprojector; and controlling depth dynamic range by controlling power ofthe sequence of light patterns being projected on the object andexposure time of a camera based on information obtained from thesequence of images captured by the camera.

In another example embodiment, the subject matter of the third exampleembodiment can optionally include that wherein the intensity informationcomprises an average intensity of a difference between intensity of asequence of images captured by the camera.

In another example embodiment, the subject matter of the third exampleembodiment can optionally include that controlling depth dynamic rangecomprises: decreasing the projector laser power if average intensitybetween a sequence of images captured by the camera is above a firstthreshold and the camera exposure is set to a preset minimal value;decreasing the exposure of the camera if the average intensity between asequence of images captured by the camera is above the first thresholdand the camera exposure is not set to the preset minimal value;increasing the camera exposure of the camera if average intensitybetween a sequence of images captured by the camera is below a secondthreshold and the projector power is set to a preset minimal value; anddecreasing the projector laser power if average intensity between asequence of images captured by the camera is below a second thresholdand the projector power is not set to the preset minimal value.

In another example embodiment, the subject matter of the third exampleembodiment can optionally include that controlling depth dynamic rangeby controlling power of the sequence of light patterns being projectedon the object and exposure time of a camera based on informationobtained from the sequence of images captured by the camera comprisesperforming automatic laser gain and automatic exposure control based onobject intensity.

In another example embodiment, the subject matter of this exampleembodiment can optionally include that performing automatic projectorgain and automatic exposure control comprises activating automaticprojector gain when object intensity associated with the objectincreases and increasing projector gain as the object intensitydecreases until the object intensity reaches a preset maximum value, andwhen the object intensity reaches the preset maximum value, thenactivating the automatic exposure control until a present minimum valueis reached after which time the automatic projector gain is activated.

Some portions of the detailed descriptions above are presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention also relates to apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

A machine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; etc.

Whereas many alterations and modifications of the present invention willno doubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that anyparticular embodiment shown and described by way of illustration is inno way intended to be considered limiting. Therefore, references todetails of various embodiments are not intended to limit the scope ofthe claims which in themselves recite only those features regarded asessential to the invention.

We claim:
 1. An apparatus comprising: a projector configured to projecta sequence of light patterns on an object; a first camera configured tocapture a sequence of images of the object illuminated with theprojected light patterns; and a controller coupled to the projector andfirst camera and operable to receive the sequence of images and performrange control by controlling power of the sequence of light patternsbeing projected on the object and exposure time of a camera based oninformation obtained from the sequence of images captured by the camera.2. The apparatus defined in claim 1 wherein the information comprisesintensity information associated with a sequence of images captured bythe camera.
 3. The apparatus defined in claim 2 wherein the intensityinformation comprises an average intensity of a difference betweenintensity of a sequence of images captured by the camera.
 4. Theapparatus defined in claim 1 wherein the controller is operable tocontrol depth dynamic range by decreasing the projector power if averageintensity between a sequence of images captured by the camera is above afirst threshold and the camera exposure is set to a preset minimal valueand decreasing the exposure of the camera if the average intensitybetween a sequence of images captured by the camera is above the firstthreshold and the camera exposure is not set to the preset minimalvalue.
 5. The apparatus defined in claim 1 wherein the controller isoperable to control depth dynamic range by increasing the cameraexposure of the camera if average intensity between a sequence of imagescaptured by the camera is below a second threshold and the projectorpower is set to a preset minimal value and decreasing the projectorpower if average intensity between a sequence of images captured by thecamera is below a second threshold and the projector laser power is notset to the present minimal value.
 6. The apparatus defined in claim 1wherein the controller is operable to control power of the sequence oflight patterns being projected on the object and exposure time of acamera based on information obtained from the sequence of imagescaptured by the camera by performing automatic projector gain andautomatic exposure control based on object intensity.
 7. The apparatusdefined in claim 6 wherein performing automatic projector gain andautomatic exposure control comprises activating automatic projector gainwhen object intensity associated with the object increases andincreasing projector gain as the object intensity decreases until theobject intensity reaches a preset maximum value, and when the objectintensity reaches the preset maximum value, then activating theautomatic exposure control until a preset minimum value is reached afterwhich time the automatic projector gain is activated.
 8. The apparatusdefined in claim 1 wherein the processing unit reconstructs depth usingtriangulation.
 9. The apparatus defined in claim 1 wherein the projectorcomprises an infrared (IR) projector, the first camera comprises an IRcamera, and the second camera comprises a red, green, and blue (RGB)camera.
 10. A method comprising: projecting a sequence of light patternson an object using a projector; capturing a sequence of images of theobject illuminated with the projected light patterns using a camera; andcontrolling depth dynamic range by controlling power of the sequence oflight patterns being projected on the object and exposure time of acamera based on information obtained from the sequence of imagescaptured by the camera.
 11. The method defined in claim 10 wherein theinformation comprises intensity information associated with a sequenceof images captured by the camera.
 12. The method defined in claim 11wherein the intensity information comprises an average intensity of adifference between intensity of a sequence of images captured by thecamera.
 13. The method defined in claim 10 wherein controlling depthdynamic range comprises decreasing the projector laser power if averageintensity between a sequence of images captured by the camera is above afirst threshold and the camera exposure is set to a preset minimal valueand decreasing the exposure of the camera if the average intensitybetween a sequence of images captured by the camera is above the firstthreshold and the camera exposure is not set to the preset minimalvalue.
 14. The method defined in claim 10 wherein controlling depthdynamic range comprises increasing the camera exposure of the camera ifaverage intensity between a sequence of images captured by the camera isbelow a second threshold and the projector power is set to a presetminimal value and decreasing the projector power if average intensitybetween a sequence of images captured by the camera is below a secondthreshold and the projector laser power is not set to the presentminimal value.
 15. The method defined in claim 10 wherein controllingdepth dynamic range by controlling power of the sequence of lightpatterns being projected on the object and exposure time of a camerabased on information obtained from the sequence of images captured bythe camera comprises performing automatic projector gain and automaticexposure control based on object intensity.
 16. The method defined inclaim 15 wherein performing automatic projector gain and automaticexposure control comprises activating automatic projector gain whenobject intensity associated with the object increases and increasingprojector gain as the object intensity decreases until the objectintensity reaches a preset maximum value, and when the object intensityreaches the preset maximum value, then activating the automatic exposurecontrol until a preset minimum value is reached after which time theautomatic projector gain is activated.
 17. The method defined in claim10 wherein the power of the sequence of light patterns comprises laserpower and the projector comprises an infrared (IR) projector.
 18. Anarticle of manufacture having one or more non-transitory computerreadable storage media storing instructions which when executed by asystem to perform a method comprising: capturing, using a camera, asequence of images of the object illuminated with projected lightpatterns on an object using a projector; and controlling depth dynamicrange by controlling power of the sequence of light patterns beingprojected on the object and exposure time of a camera based oninformation obtained from the sequence of images captured by the camera.19. The article of manufacture defined in claim 18 wherein the intensityinformation comprises an average intensity of a difference betweenintensity of a sequence of images captured by the camera.
 20. Thearticle of manufacture defined in claim 18 wherein controlling depthdynamic range comprises: decreasing the projector laser power if averageintensity between a sequence of images captured by the camera is above afirst threshold and the camera exposure is set to a preset minimalvalue; decreasing the exposure of the camera if the average intensitybetween a sequence of images captured by the camera is above the firstthreshold and the camera exposure is not set to the preset minimalvalue; increasing the camera exposure of the camera if average intensitybetween a sequence of images captured by the camera is below a secondthreshold and the projector power is set to a preset minimal value; anddecreasing the projector laser power if average intensity between asequence of images captured by the camera is below a second thresholdand the projector power is not set to the preset minimal value.